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INTRODUCTION 


In  military  operations,  tasks  will  often  need  to  be  accomplished  through  the  joint  efforts  of  several 
individuals.  This  may  include  multiple  individuals  involved  in  a  common  effort  or  in  a  coordinated 
activity  (such  as  a  joint  strike  mission).  The  critical  role  of  teamwork  in  accomplishing  mission- 
oriented  tasks  and  goals  in  today’s  military  is  unquestionable  (Salas,  Dickenson,  Converse,  & 
Tannenbaum,  1992).  Today,  mission  success  demands  the  coordinated  effort  of  those  individuals 
who  have  become  specialists,  as  each  one  of  them  only  has  a  piece  of  the  puzzle.  Only  by 
communicating  and  coordinating  their  efforts  can  the  team  be  successful  in  a  timely  fashion.  The 
tremendous  volume  and  complexity  of  data  that  the  team  must  encounter  and  process  to  construct  an 
accurate  tactical  picture  is  overwhelming.  This  process  becomes  even  more  overwhelming  with  the 
increased  use  of  distributed  interactive  networks  of  teams  with  members  who  have  a  variety  of 
duties  and  training.  Collaborative  decision  making  may  be  required  for  effective  planning 
operations,  for  sharing  updates  on  situational  changes  (e.g.  enemy  activities),  or  for  creating 
effective  responses  to  such  changes,  and  a  wide  variety  of  problem  solving  tasks.  Effectively 
coordinating  the  activities  and  decision  making  of  these  multiple  personnel  remains  a  challenge, 
however. 

This  research  is  directed  towards  developing  effective  tools  and  techniques  for  assisting  individuals 
in  achieving  a  high  level  of  shared  situation  awareness  (SA)  when  involved  in  collaborative 
decision  making  tasks  such  as  those  to  be  found  in  future  military  operations.  This  effort  will  be 
focused  around  the  problem  of  providing  shared  situation  awareness.  Teams  of  individuals  can  be 
defined  in  terms  of  common  goals,  interdependence  of  task,  and  specific  roles  towards  meeting  the 
shared  goal  (Salas,  et  al.,  1992).  In  order  to  perform  their  tasks,  each  member  of  the  team  (which 
may  be  a  standing  team  or  may  be  formed  ad  hoc  for  solving  some  problem)  needs  to  have  a  certain 
level  of  SA.  Examining  situation  awareness  as  it  exists  within  teams  and  between  teams  that  are 
involved  in  achieving  a  common  goal  lends  an  important  perspective  for  the  determination  of 
system  designs  that  support  the  complex  inter-related  activities  of  these  teams.  Team  SA  has  been 
defined  as  "the  degree  to  which  every  team  member  possess  the  SA  required  for  his  or  her 
responsibilities"  (Endsley,  1995).  The  degree  to  which  team  members  possess  a  shared 
understanding  of  the  situation  with  regard  to  their  shared  SA  requirements  is  extremely  important 
for  developing  effective  team  performance. 

Developing  shared  SA  within  a  team  and  between  teams  can  be  extremely  challenging,  especially 
where  those  teams  are  distributed  in  terms  of  space,  time,  or  physical  barriers.  This  process  has  been 
described  in  a  model  of  team  SA  as  coming  about  as  a  function  of  four  components  (Endsley  & 
Jones,  1997): 

(1)  Shared  SA  Requirements  -  the  degree  to  which  the  team  members  know  which  information 
needs  to  be  shared,  including  their  higher  level  assessments  and  projections  (which  are  usually 
not  otherwise  available  to  fellow  team  members),  and  information  on  team  members'  task  status 
and  current  capabilities. 

(2)  Shared  SA  Devices  -  the  devices  available  for  sharing  this  information,  which  can  include  direct 

communication  (both  verbal  and  nonverbal),  shared  displays,  or  a  shared  virtual  environment. 
As  nonverbal  communications  and  a  shared  environment  are  usually  not  available  in 
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distributed  teams,  this  places  far  more  emphasis  on  verbal  communication  and  technologies  for 
creating  shared  information  displays. 

(3)  Shared  SA  Mechanisms  -  the  degree  to  which  team  members  possess  mechanisms,  such  as 
shared  mental  models,  which  support  their  ability  to  interpret  information  in  the  same  way  and 
make  accurate  projections  regarding  each  other’s  actions.  The  possession  of  shared  mental 
models  can  greatly  facilitate  communication  and  coordination  in  team  settings. 

(4)  Shared  SA  Processes  -  the  degree  to  which  team  members  engage  in  effective  processes  for 
sharing  SA  information  which  may  include  a  group  norm  of  questioning  assumptions,  checking 
each  other  for  conflicting  information  or  perceptions,  setting  up  coordination  and  prioritization 
of  tasks,  and  establishing  contingency  planning  among  others. 

Recent  research  has  been  conducted  to  explore  this  model  of  team  SA.  Bolstad  and  Endsley  (1998) 
examined  the  use  of  shared  mental  models  and  shared  displays  (components  2  and  3  of  the  team  SA 
model)  as  a  means  of  enhancing  team  situation  awareness.  The  study  demonstrated  that  when 
members  of  a  team  are  dependent  on  each  other  for  successful  performance,  the  presence  of  a 
shared  mental  model  helped  to  improve  overall  team  performance  as  expected.  The  mechanism 
whereby  the  shared  displays  aided  performance  was  not  direct  as  expected.  Teams  actually 
performed  worse  with  a  shared  display;  however,  a  residual  effect  was  seen  in  later  trials.  After  the 
shared  displays  were  removed,  the  teams  outperformed  both  their  prior  performance  and  other 
teams  that  received  the  non-shared  display  condition  followed  by  the  shared  display  condition. 
While  the  presence  of  shared  displays  slowed  team  performance  in  this  task,  most  likely  due  to 
extra  attention  demands,  they  also  provided  for  the  development  of  shared  mental  models  that 
greatly  enhanced  performance  even  after  the  shared  displays  were  removed.  The  combination  of  no 
shared  displays  and  no  mental  model  was  highly  detrimental  to  performance.  Teams  who 
experienced  this  condition  first  were  unable  to  ever  develop  very  good  performance. 

In  the  present  study,  we  sought  to  expand  on  the  previous  research  by  examining  the  use  of 
abstracted  shared  displays.  In  the  previous  research,  the  shared  display  condition  provided  each 
team  member  with  a  complete  replicate  of  the  other  team  member’s  display.  This  may  not  be  the 
best  way  to  provide  a  shared  display  of  information  to  team  members,  however.  An  abstracted 
shared  display  instead  provides  only  the  critical  information  from  the  other  team  member’s  display, 
based  on  an  analysis  of  shared  information  requirements  (Endsley  &  Jones,  1997).  It  is 
hypothesized  that  the  use  of  abstracted  shared  displays  might  help  to  build  team  SA  without 
imposing  the  extra  workload  observed  with  the  use  of  the  full  shared  displays. 

Secondly,  we  wished  to  further  explore  the  issues  associated  with  workload  level  and  its  affect  on 
both  team  interaction  and  on  the  use  of  the  shared  displays  in  the  development  of  team  strategies  for 
performing  the  task.  In  addition,  the  Theater  Defense  team  task  that  was  used  for  the  study  was 
enhanced  to  create  a  more  complex  task.  The  Air  Commander  was  given  two  different  missiles 
types,  each  more  effective  for  different  types  of  aircraft  in  the  simulation.  Three  AWACS-like 
(Airborne  Warning  and  Control  System)  aircraft  were  added  to  the  simulation  so  that  the  reliability 
of  the  sensors  varied  over  time,  depending  on  the  AW  ACS  position.  This  created  a  more  complex 
task  and  furthered  the  degree  of  interaction  required  by  the  two  team  members. 
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METHOD 


Participants 

Thirty-eight  participants  served  as  paid  subjects  for  this  research.  These  participants  were  recruited 
from  a  large  Southern  university.  Two  participants  (one  team)  were  dropped  from  the  analysis  due 
to  their  low  overall  performance  (less  than  three  standard  deviations  below  the  mean  on 
performance  measures).  The  remaining  36  participants  were  tested  in  pairs  for  a  total  of  18  teams. 
The  participants  (mean  age  =  23.19  years;  range  18-43  years)  comprised  21  men  and  15  women 
with  an  average  of  15.31  years  of  formal  education.  All  participants  indicated  they  had  normal  or 
better  vision.  The  participants  had  an  average  of  6.93  years  of  computer  experience  and  were 
familiar  with  the  general  operation  of  a  personnel  computer. 

Design 

Two  factors  served  as  independent  variables  in  the  study:  Workload  and  Shared  Display  Type. 

(1)  Workload  Level:  Three  workload  levels  (3,  6,  and  9)  were  determined  based  on  the  maximum 
number  of  aircraft  on  the  display  at  any  one  time.  These  loads  were  selected  as  a  “low,” 
“medium,”  and  “high”  workload  level,  based  on  previous  research.  Workload  was  a  within 
team  manipulation. 

(2)  Shared  Display  Types:  Non-Shared  Displays  (verbal  interaction  only)  which  served  as  a  control 
condition.  Full  Shared  Displays,  and  Abstracted  Shared  Displays  were  examined  as  the  three 
conditions  of  this  independent  variable.  Shared  Display  Type  was  a  between  team  manipulation. 

Six  teams  completed  the  study  in  each  of  the  three  Shared  Display  conditions,  as  shown  in  Table  1. 
The  order  of  presentation  of  workload  level  was  counter-balanced  across  teams.  All  teams 
completed  three  6-minute  trials  at  each  workload  level,  for  a  total  of  nine  trials  per  team. 


Table  1.  Experimental  Design 


Workload  Level 
Order 

No 

Shared  Display 

Full  Shared 
Display 

Abstracted 
Shared  Display 

3,6,9 

Team  1 

Team  2 

Team  3 

3,9,6 

Team  4 

Team  5 

Team  6 

6,3,9 

Team  7 

Team  8 

Team  9 

6,9,3 

Team  10 

Team  1 1 

Team  12 

9,3,6 

Team  13 

Team  14 

Team  15 

9,6,3 

Team  16 

Team  17 

Team  18 

The  effects  of  the  independent  variables  on  the  performance  of  the  two  team  members  in  the 
Theatre  Defense  task  were  examined  as  dependent  variables.  The  processing  outcome  for  each 
aircraft  (destroyed,  passed  through,  or  collided),  time  to  process  a  target,  and  reward  and  penalty 
points  were  recorded  for  the  Air  Commander.  The  time  to  make  an  identification,  correctness  of 
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identification,  and  the  use  of  the  Air  Commander’s  prioritization  order  were  recorded  for  the 
Intelligence  Officer.  Verbal  communications  were  also  recorded  between  team  members,  including 
both  inquiries  and  information  provision  to  the  other  team  member. 


Procedure 

Teams  were  tested  one  at  a  time.  Team  members  were  given  a  handout  describing  their  task  within 
the  Theatre  Defense  team  task.  See  Appendix  A  for  the  Intelligence  Officer’s  job  description  for 
Non-Shared  and  Full  Shared  Display  conditions  and  Appendix  B  for  the  Abstracted  Shared  Display 
condition.  See  Appendix  C  for  the  Air  Commander’s  job  description  for  Non-Shared  and  Full 
Shared  Display  conditions  and  Appendix  D  for  the  Abstracted  Shared  Display  condition.  In  the 
Abstract  Shared  Display  condition,  information  regarding  the  additional  abstracted  displays  were 
included  in  the  instructions. 

Teams  in  the  Non-Shared  Display  condition  were  not  given  any  information  regarding  the  other 
team  member’s  task.  During  the  task,  team  members  were  seated  side  by  side,  but  they  were 
separated  by  a  barrier  and  thus  could  not  view  each  other  or  the  other  team  member’s  displays. 
Team  members  in  the  Full  Shared  Display  condition  were  seated  side  by  side  with  their  computer 
monitors  approximately  six  inches  apart.  This  placement  allowed  them  the  ability  to  view  the  other 
team  member’s  computer  screen  while  performing  the  task. 

After  reading  the  instructions  and  at  the  completion  of  each  trial,  participants  were  given  time  to  ask 
questions  to  clarify  their  tasks.  In  all  conditions,  team  members  could  communicate  with  one 
another  verbally  during  the  task  and  were  encouraged  to  determine  what  the  other  team  member 
was  doing,  as  well  as  work  together  on  a  joint  strategy  to  improve  performance  during  the  task. 
Teams  completed  two  6-minute  practice  trials  followed  by  nine  6-minute  test  trials  (three  at  each 
workload  level).  Participants  were  given  a  15  minute  break  after  completing  the  first  three  test  trials. 


Task 

The  Theatre  Defense  program  used  in  the  previous  research  (Bolstad  &  Endsley,  1998)  was 
modified  for  this  study  to  include  the  Abstract  Shared  Display  condition,  varying  sensor  reliability, 
and  two  different  missile  types,  as  described  in  more  detail  below.  The  Theatre  Defense  microworld 
was  written  in  Microsoft  Visual  Basic.  It  was  based  on  Multitask,  a  single  person  control  task 
created  by  Kaber  and  Endsley  (1997).  Theatre  Defense  was  hosted  on  two  separate  Pentium  based 
workstations  that  were  connected  by  an  Ethernet  LAN.  Data  for  each  team  member  was  recorded 
by  the  workstation  computers. 

Theatre  Defense  incorporates  activities  by  two  individuals:  an  Intelligence  Officer  and  an  Air 
Commander  who  each  have  separate,  but  inter-related  tasks.  The  two  team  members  work  at 
separate  workstations,  connected  by  an  Ethernet  LAN.  The  role  of  the  Air  Commander  is  to  protect 
the  home  base  from  incoming  aircraft.  Targets  (designated  by  blank  boxes)  appear  on  the  radar 
screen  moving  towards  a  central  point  (the  home  base),  as  shown  in  Figure  1.  The  Air  Commander 
must  prioritize  these  targets  (based  on  range  and  speed)  and  request  information  from  the 
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Intelligence  Officer  on  their  identity  and  mission  priority  as  well  as  provide  AW  AC  location 
information,  which  determines  sensor  reliability.  Once  an  identification  has  been  received  from  the 
Intelligence  Officer,  the  Air  Commander  processes  the  targets  accordingly.  The  Air  Commander 
must  choose  which  targets  to  destroy  (based  on  range,  speed,  and  penalty/reward  points)  and  which 
to  let  through  to  the  home  base  (such  as  friendlies). 

Targets  included  fighter,  bomber,  and  transport  aircraft  of  either  friendly  or  enemy  designations, 
making  for  a  possibility  of  six  categories  with  a  total  of  16  different  aircraft  types.  Points  were 
assigned  to  each  category  representing  the  reward  points  for  destroying  the  aircraft  and  penalty 
points  for  allowing  the  aircraft  to  get  through  to  the  home  base.  Reward  and  penalty  points  for  each 
category  were  based  on  the  mission  relevance  and  lethality  of  the  aircraft  type.  Friendly  aircraft  had 
a  zero  penalty  for  getting  through  to  home  base  and  a  negative  reward  associated  with  destroying 
them.  In  addition  to  landing  or  destroying  an  aircraft,  it  was  possible  for  some  targets  to  collide 
with  each  other,  resulting  in  the  reward  and  penalty  points  associated  with  both  aircraft  to  be 
recorded.  Thus,  the  Air  Commander  needed  to  correctly  destroy  enemy  aircraft  that  would  conflict 
with  friendly  aircraft  before  such  a  collision  occurred. 


Figure  1.  Air  Commander  Workstation’s  Screen 
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If  the  Air  Commander  decided  to  attack  a  target,  he/she  was  required  to  choose  which  missile  to 
launch  for  the  attack.  There  were  two  types  of  simulated  missiles  available:  a  Sparrow  which  is 
smaller  and  faster  and  used  for  destroying  fighters  and  an  AMRAAM  (advanced  medium  range  air 
to  air  missile)  which  is  a  long  range  radar  guided  missile  used  for  destroying  bombers  and 
transports.  The  left  mouse  button  launched  Sparrows  and  the  right  mouse  button  launched 
AMRAAMs.  If  the  appropriate  missile  was  launched,  the  plane  was  destroyed,  however,  if  the 
incorrect  missile  was  launched  for  the  targeted  aircraft,  the  chances  that  the  plane  would  be 
destroyed  were  less  than  50%. 

The  Intelligence  Officer  was  supplied  with  a  list  of  targets  and  the  identifications  provided  by 
several  different  sensors,  as  shown  in  Figure  2.  Upon  a  request  from  the  Air  Commander,  the 
Intelligence  Officer  needed  to  select  the  sensor  information  for  that  target  and  make  a  designation  of 
the  target  aircraft  type  and  category.  Three  sensors  (A,  B,  and  C)  mounted  on  mobile  AW  ACS 
aircraft  were  provided.  Participants  were  instructed  that  the  sensor  reliability  rates  were  dependent 
upon  their  location  and  could  either  be  75%,  50%,  or  25%.  The  Intelligence  Officer  could  receive 
the  current  sensor  reliability  by  requesting  AWACS  locations  from  the  Air  Commander  or  by  the 
Air  Commander  providing  it  spontaneously. 
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Figure  2.  Intelligence  Officer’s  Workstation  Screen 
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To  represent  issues  present  in  the  real  world,  the  information  from  the  three  sensors  was  either 
consistent  (1/3  of  the  cases),  partially  missing  (1/3  of  the  cases),  or  dissonant  (1/3  of  the  cases).  The 
distribution  of  the  dissonant  and  missing  information  was  distributed  across  sensor  types  and 
reliabilities.  For  instance,  in  some  cases  two  sensors  indicated  a  friendly  and  one  an  enemy,  and  in 
other  cases  two  indicated  an  enemy  and  one  a  friendly.  Which  sensor  provided  the  dissonant 
information  was  also  counter-balanced  across  cases  to  provide  for  information  on  the  effect  of 
sensor  reliability  as  well  as  sensor  confirmation  in  making  identification  decisions. 

Once  the  Intelligence  Officer  made  a  decision  on  aircraft  identification,  this  information  was  passed 
to  the  Air  Commander  who  saw  the  target  change  colors  (indicating  its  category)  and  the  reward 
and  penalty  points  associated  with  the  target  (based  on  its  type  of  classification).  This  information 
was  dependent  on  the  Intelligence  Officer’s  identification  and  therefore  may  or  may  not  have  been 
correct.  However,  the  points  assigned  when  the  target  was  processed  (landed,  destroyed,  or 
collided)  were  based  on  the  actual  identification  of  the  target.  Feedback  was  provided  to  the 
Intelligence  Officer  only  after  the  target  was  destroyed  or  allowed  to  pass  through  to  the  home  base, 
as  in  the  real  world  (e.g.  F-16  passed  through;  F-16  destroyed).  The  Air  Commander  was  also 
provided  with  the  running  point  total.  Learning,  therefore,  was  able  to  take  place  allowing  the  team 
members  to  develop  effective  strategies  and  decision  behaviors. 

The  pace  of  the  task  was  such  that  to  maximize  points,  both  team  members  needed  to  be  very 
strategic  about  how  they  prioritized  and  processed  targets.  Otherwise,  a  significant  number  of 
enemy  aircraft  would  penetrate  the  air  defense  and  strike  the  home  base,  or  a  significant  number  of 
friendly  aircraft  would  fall  prey  to  fratricide. 

In  the  Non-Shared  Display  condition,  each  officer  had  only  a  limited  amount  of  information  from 
the  other  team  member  in  addition  to  their  own  display.  The  Intelligence  Officer  saw  only  the 
prioritization  list  provided  by  the  Air  Commander  and  the  Air  Commander  saw  only  the  resultant 
classification  provided  by  the  Intelligence  Officer.  Any  other  exchange  of  information  had  to  occur 
verbally. 

In  the  Full  Shared  Display  condition,  each  officer  also  saw  all  the  display  information  of  the  other 
officer.  Therefore,  the  Intelligence  Officer  also  saw  a  picture  of  the  Air  Commander’s  radar  display 
which  it  was  hypothesized  would  help  him/her  better  prioritize  and  anticipate  the  prioritization  of 
the  targets  to  be  identified.  It  also  allowed  the  Intelligence  Officer  to  directly  assess  sensor 
reliability  based  on  AW  ACS  position,  rather  than  having  to  request  this  information  from  the  Air 
Commander. 

The  displays  were  modified  in  the  Abstracted  Shared  Display  condition  to  provide  each  officer  with 
all  the  needed  information  from  the  other  team  member’s  display,  based  on  an  analysis  of  shared 
information  requirements.  In  this  condition,  the  Air  Commander  also  received  sensor  data  in  the 
lower  left  comer  of  the  screen  as  shown  in  Figure  3.  This  information  was  sent  over  to  the  Air 
Commander  through  a  datalink  capability  after  the  Intelligence  Officer  identified  each  target. 
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Figure  3.  Air  Commander  Workstation’s  Screen  -  Abstract  Shared  Display 


In  the  Abstracted  Shared  Display  condition,  the  Intelligence  Officer  also  received  information  on 
target  proximity  from  home  base  in  the  lower  right  corner  of  the  screen,  as  shown  in  Figure  4.  The 
targets  are  listed  from  top  to  bottom  with  the  top  target  being  the  closest  to  home  base  and  the 
bottom  target  being  the  farthest.  To  the  left  of  this  display  the  sensor  reliability  information  was 
shown,  followed  by  the  point  totals. 

It  was  hypothesized  that  the  Abstracted  Shared  Displays  would  help  both  officers  better  perform 
their  tasks  by  enhancing  shared  SA  on  this  team  task.  It  was  hypothesized  that  the  Full  Shared 
Display  condition  would  lower  team  performance,  as  was  observed  in  the  previous  study.  It  was 
also  hypothesized  that  the  Full  and  Abstracted  Shared  Display  conditions  would  lead  to  lower  levels 
of  team  communication  as  less  information  would  need  to  be  verbally  transferred  from  one  officer 
to  another.  Finally,  it  was  hypothesized  that  the  task  workload  level  would  interact  with  all  of  these 
findings.  Specifically,  we  expected  that  both  the  Full  and  Abstracted  Shared  Displays  would  have 
more  of  an  effect  under  high  workload  conditions. 
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Figure  4.  Intelligence  Officer’s  Workstation  Screen  -  Abstract  Shared  Display 


RESULTS 

Scoring 

Data  was  collected  from  both  participants  during  the  trials.  In  order  to  facilitate  data  analysis  (as 
each  team  produced  more  than  650  separate  target  entries),  summary  files  were  created  containing 
means  for  the  variables  of  interest  for  each  of  the  nine  trials  for  each  team.  Mean  penalty  points, 
mean  reward  points,  and  mean  decision  time  to  expiration,  collision,  or  attack  were  calculated  for 
each  trial  for  each  team.  Mean  time  from  target  information  request  to  classification,  mean  target 
viewing  time,  the  percentage  of  targets  that  were  requested  by  the  Air  Commander  at  time  of 
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classification,  and  the  percentage  of  targets  that  were  at  the  top  of  the  Air  Commander’s  request  list 
at  the  time  of  classification  were  also  calculated  for  each  trial  for  each  team. 

Three  sets  of  analysis  were  conducted.  The  first  examined  the  effects  of  Shared  Display  Type  and 
Workload  Level  on  team  performance.  The  second  analysis  examined  the  influence  of  sensor 
performance,  in  particular  dissonant  data,  on  decision  making.  The  final  analysis  examined  the 
effects  of  Shared  Display  Type  and  Workload  Level  on  the  teams’  verbal  communications.  All 
analyses  were  performed  using  analysis  of  variance  (ANOVA).  Tukey  tests  were  used  for  post-hoc 
analysis.  We  used  an  alpha  level  of  .05  for  all  analyses. 


Shared  Display  Type  and  Workload 

Penalty  points,  reward  points,  decision  making  time,  percentage  of  targets  classified  from  the 
request  list,  and  percentage  of  targets  classified  from  the  top  of  the  Air  Commander’s  request  list 
were  each  examined  using  an  ANOVA  which  included  Shared  Display  Type,  Workload  Level,  and 
their  interaction. 

Penalty  Points  and  Reward  Points 


Both  the  reward  points,  F(2,153)  =  8.68,  p  <  .001,  and  penalty  points,  F(2,153)  =  5.945,  p  -  .003, 
significantly  varied  across  the  three  display  conditions.  Reward  points  were  lowest  in  the  Full 
Shared  Display  condition.  The  Non-Shared  and  Abstracted  Shared  Displays  were  similar  in  terms  of 
reward  points,  as  shown  in  Figure  5.  Penalty  points  were  highest  in  the  Full  Shared  condition. 
Penalty  points  with  the  Abstracted  Shared  Display  conditions  were  the  least. 
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Figure  5.  Mean  Reward  and  Penalty  Points  by  Shared  Display  Type 


11 


Thus,  as  compared  to  the  Non-Shared  Display  condition  (control),  teams  with  the  Full  Shared 
Displays  obtained  both  fewer  reward  points  and  more  penalty  points  for  overall  worse  performance. 
This  confirms  the  findings  of  the  previous  study  (Bolstad  &  Endsley,  1998).  In  comparison,  teams 
with  the  Abstracted  Shared  Displays  obtained  about  the  same  level  of  reward  points  as  the  control 
condition,  without  an  increase  in  penalty  points.  In  fact,  their  penalty  points  were  slightly  lower 
(although  not  significantly).  The  Abstracted  Shared  Display  significantly  out  performed  the  Full 
Shared  Display  on  both  measures. 

As  expected,  both  reward,  F( 2,  153)  =  125.335,  p  <  .001,  and  penalty  points,  F( 2,  153)  =  183.682,  p 
<  .001,  increased  significantly  from  Workload  Level  three  to  Workload  Level  nine  as  shown  in 
Figure  6.  The  number  of  reward  and  penalty  points  accrued  is  highly  dependent  on  the  number  of 
aircraft  appearing  during  the  test.  The  higher  the  Workload  Level,  the  greater  the  number  of  planes 
and  hence  the  opportunities  to  accrue  points.  To  better  examine  this  variable,  the  reward  and 
penalty  points  were  normalized.  The  normalized  score  for  each  team  was  calculated  as  the 
percentage  of  reward  or  penalty  points  achieved  by  the  team  as  a  function  of  the  amount  possible 
during  the  trial. 
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Level  3  Level  6  Level  9 


Figure  6.  Mean  Reward  and  Penalty  Points  by  Workload  Level 
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An  analysis  of  these  normalized  scores  showed  that  both  the  normalized  reward  points,  F( 2,  153)  = 
3.17,  p  =  .045,  and  normalized  penalty  points  F(2,  153)  =  7.359,  p  =  .001,  were  effected  by 
Workload  Level,  as  shown  in  Figure  7.  They  were  significantly  higher  at  a  Workload  Level  of  nine 
and  lower  at  a  Workload  Level  of  three,  as  before.  Thus,  the  changes  in  performance  at  different 
Workload  Levels  were  not  strictly  due  to  the  changed  opportunities  for  scoring,  but  due  to  other 
changes  in  the  teams’  ability  to  carry  out  the  tasks  under  the  different  loading  conditions.  At  the 
High  Workload  Level,  the  teams  began  “shooting  at  everything,”  creating  both  proportionately 
higher  rewards  and  higher  penalties.  At  the  Low  Workload  Level,  they  were  able  to  be  more 
careful  and  minimized  penalties. 
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Figure  7.  Mean  Adjusted  Reward  and  Penalty  Points  by  Workload  Level 


The  interaction  of  Shared  Display  Type  and  Workload  Level  was  significant  for  both  reward  points, 
F( 4,  153)  =  2.838,  p  =  .026,  and  penalty  points,  F(4,  153)  =  3.407,  p  =  .01 1.  At  the  Low  Workload 
Level,  the  number  of  reward  and  penalty  points  did  not  vary  significantly  across  Shared  Display 
Type  conditions.  This  indicates  that  at  Low  Workload  Levels,  the  presence  or  type  of  Shared 
Displays  does  not  present  either  a  problem  or  a  significant  boost  to  performance.  This  was  not  true 
for  the  Moderate  and  High  Workload  Levels,  however. 
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As  shown  in  Figure  8,  while  reward  points  were  higher  in  the  High  Workload  conditions,  the 
increases  were  less  for  the  Full  Shared  Display  condition  than  for  the  Non-Shared  and  Abstracted 
Shared  Display  conditions.  In  fact,  in  the  Full  Shared  Display  condition,  performance  eventually 
plateaued  at  Workload  Levels  six  and  nine,  at  a  performance  level  slightly  lower  than  that  observed 
for  the  Non-Shared  Display  condition  at  Workload  Level  six.  Performance  with  the  Abstracted 
Display  condition  did  not  show  this  problem,  however,  with  the  reward  points  essentially  equal  to 
that  in  the  Non-Shared  Display  condition. 

As  shown  in  Figure  9,  the  penalty  points  were  also  higher  in  the  Full  Shared  Display  condition  at 
the  Highest  Workload  Level.  Penalty  points  were  actually  lower  with  the  Abstracted  Shared 
Display  at  the  Moderate  and  High  Workload  Levels  as  compared  to  the  two  other  Shared  Display 
conditions.  Thus,  it  can  be  seen  that  the  Full  Shared  Display  created  a  particular  problem  at  the 
Higher  Workload  Levels.  The  Abstracted  Shared  Display,  on  the  other  hand,  allowed  penalty 
reduction  at  the  Higher  Workload  Levels. 
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Figure  8.  Mean  Reward  Points  by  Shared  Display  Type  and  Workload  Level 


14 


Non-Shared  Full  Shared  Abstracted  Shared 
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Figure  9.  Mean  Penalty  Points  by  Shared  Display  Type  and  Workload  Level 


Decision  Time 


Shared  Display  Type  also  significantly  effected  team  decision  times.  When  the  Intelligence 
Officers  were  given  either  the  Full  Shared  Display  or  Abstracted  Shared  Display,  they  were 
significantly  slower  in  making  target  classifications,  F{ 2,  143)  =  8.821,  p  <  .001,  as  can  be  seen  in 
Figure  10.  The  Intelligence  Officer  may  have  been  slowed  down  by  the  presence  of  the  additional 
target  information  from  either  the  Full  Shared  Display  or  Abstracted  Shared  Display. 

Shared  Display  Type  also  significantly  effected  the  time  it  took  the  Air  Commander  to  attack,  F{ 2, 
153)  =  7.359  p  <  .001,  shown  in  Figure  10.  They  took  the  longest  time  to  make  a  decision  with  the 
Abstracted  Shared  Display,  most  likely  due  to  the  need  to  consider  more  information.  Surprisingly, 
this  was  not  the  case  with  the  Full  Shared  Display.  This  may  be  because  the  Air  Commanders  were 
not  using  the  information  on  the  Full  Shared  Display  in  many  cases  due  to  overload.  The  greater 
likelihood  of  using  the  information  provided  in  the  Abstracted  Display  condition  was  realized  in  the 
better  performance  for  these  teams,  even  though  the  Air  Commander  took  longer  to  make  attack 
decisions. 
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Figure  10.  Mean  Decision  Time  by  Shared  Display  Condition 


As  the  Workload  Level  increased,  the  Intelligence  officer  took  significantly  less  time  to  make 
aircraft  classifications,  F(2,143)  =  45.19,  p  <  .001.  That  is,  as  they  had  more  aircraft  to  deal  with, 
they  made  each  decision  more  quickly  since  they  had  to  make  more  decisions  in  the  same  time 
period.  The  Air  Commander  took  longer  to  make  the  decision  to  attack  aircraft  at  Higher  Workload 
Levels,  F(2,146)  =  21.437,  p  <  .001,  as  shown  in  Figure  11.  Although  they  had  slightly  more  time 
due  to  the  faster  target  classification  of  the  Intelligence  Officer,  they  also  had  many  more  aircraft  to 
contend  with.  The  long  decision  time  by  the  Air  Commanders  at  the  Higher  Workload  Levels  most 
likely  reflects  the  fact  that  as  they  got  more  “loaded  up,”  it  took  them  longer  to  get  to  each  aircraft. 
They  could  not  process  each  one  as  it  was  classified  on  their  screen.  In  addition,  under  High 
Workload,  they  often  started  using  the  wrong  missiles,  which  slowed  down  their  performance. 

Finally,  there  was  also  a  significant  Shared  Display  Type  by  Workload  Level  interaction  for  the 
time  it  took  the  Intelligence  Officer  to  classify  the  aircraft,  F(4,143)  =  3.221,  p  <  .001.  The 
interaction  was  due  to  significantly  longer  classification  times  at  the  Lowest  Workload  Level  for  the 
Full  Shared  Display  and  Abstract  Shared  Display  conditions.  This  effect  is  shown  in  Figure  12. 
Under  the  Low  Workload  Level,  participants  really  used  the  time  available  to  acquire  extra 
information  from  the  Shared  Displays.  This  effect  was  less  evident  at  the  Moderate  and  High 
Workload  Levels. 
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Figure  11.  Mean  Decision  Time  by  Workload  Level 
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Figure  12.  Mean  Classification  Time  by  Display  Type  and  Workload  Level 
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Classification  of  Priority  Targets 


The  degree  to  which  the  Intelligence  Officer  classified  targets  that  the  Air  Commander  had 
requested  was  examined  as  an  index  of  the  degree  to  which  the  two  participants  were  operating  as  a 
team.  The  percentage  of  targets  classified  that  were  on  the  request  list  was  significantly  effected  by 
Shared  Display  Type,  F(2,149)  =20.66,  p  <  .001,  and  Workload  Level,  F(2,149)  =14.51 1,  p  <  .001. 
These  effects  are  shown  in  Figure  13. 

Over  50  percent  of  the  targets  categorized  had  been  requested  by  the  Air  Commander  (on  the 
request  list)  in  the  Full  Shared  Display  condition  and  fewer  than  20  percent  had  been  requested  in 
the  other  two  conditions.  While  it  is  possible  that  participants  may  have  requested  the  target 
identifications  verbally,  the  analysis  of  verbal  interactions  does  not  bear  out  this  possibility  (see 
page  19).  It  therefore  indicates  very  limited  team  interaction  in  the  Non-Shared  Display  condition. 
The  provision  of  the  Full  Shared  Displays  greatly  increased  team  coordination,  as  evidenced  by  this 
factor,  although  at  a  decrement  to  performance.  The  Abstracted  Shared  Display  condition  did  not 
show  such  an  effect,  however.  It  is  most  likely  that  in  this  condition,  they  were  able  to  get  the 
information  participants  needed  from  the  Abstracted  Shared  Displays  directly,  and  therefore  had 
less  need  to  communicate  this  information  formally.  Unlike  in  the  Non-shared  Display  condition, 
however,  overall  team  performance  was  higher. 

Participants  also  categorized  the  largest  percentage  of  requested  targets  at  the  lowest  workload 
level.  Air  Commanders  were  less  likely  to  request  the  needed  information  at  the  higher  workload 
levels  or  Intelligence  Officers  were  less  likely  to  attend  to  these  requested  prioritizations.  Thus  the 
team  members  operated  more  independently  in  these  conditions. 


Non-  Full  Shared  Abstracted  Level  3  Level  6  Level  9 

Shared  Displays  Shared 
Displays  Displays 


Figure  13.  Percent  of  Targets  Categorized  that  were  Requested  by  the  Air  Commander 
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Similarly,  the  percentage  of  targets  classified  that  were  at  the  top  of  the  request  list  (the  highest 
priority)  was  significantly  affected  by  Shared  Display  Type,  F(2,149)  =  10.832  p  <  .001,  and 
Workload  Level,  F{ 2,  149)  =  17.529,  p  <  .001.  There  was  also  a  significant  Display  Type  by 
Workload  Level  interaction  for  this  variable,  F(4,149)  =  2.674,  p  =  .034,  shown  in  Figure  14. 
More  targets  were  classified  from  the  top  of  the  request  list  in  the  Full  Shared  Display  condition,  in 
particular  at  the  lowest  workload  level,  than  with  any  other  Shared  Display  Type  or  Workload  Level 
combination. 


Non-Shared  Full  Shared  Abstracted  Shared 

Displays  Displays  Displays 


Figure  14.  Percent  of  Targets  Categorized  from  the  Top  of  the  Request  List 


Verbal  Communications  Analysis 

The  second  analysis  focused  on  the  effects  of  Shared  Display  Type  and  Workload  Level  on  the 
verbal  communication  that  occurred  between  team  members.  During  the  test,  a  tally  was  kept  of  the 
type  of  verbal  exchange  occurring  between  team  members.  For  this  analysis,  communications  were 
recorded  separately  for  three  categories  (1)  information  provided  from  the  Intelligence  Officer  to  the 
Air  Commander,  (2)  information  provided  from  the  Air  Commander  to  the  Intelligence  Officer,  and 
(3)  joint  communications  that  involved  discussions  between  the  two  team  members.  In  addition, 
within  each  of  these  categories,  requests  and  provisions  of  aircraft  identification  were  tallied 
separately  from  other  communications  (as  the  former  could  be  expected  to  change  as  a  function  of 
workload  level),  creating  a  total  of  6  communication  categories. 
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The  requesting  of  information  about  aircraft  identification  by  the  Air  Commander  varied  by 
Workload  Level,  F(2,45)  =  6.518,  p=.003.  As  shown  in  Figure  15,  they  needed  to  request 
significantly  more  aircraft  identifications  as  the  number  of  aircraft  increased.  There  were  no 
changes  in  this  variable  as  a  function  of  Shared  Display  Type.  There  were  also  no  changes  in  the 
verbal  provision  of  aircraft  identification  by  the  Intelligence  Officer  or  joint  communications  about 
target  identification  as  a  function  of  Workload  Level  or  Shared  Display  Type. 

In  terms  of  other  information  communications,  there  was  a  significant  change  in  the  degree  to 
which  the  Air  Commander  provided  information  to  the  Intelligence  Officer  in  the  two  Shared 
Display  conditions,  F(2,45)  =  13.78,  p  =  .000.  As  shown  in  Figure  16,  there  was  information  (such 
as  sensor  reliability)  directly  from  the  shared  displays  and  required  less  verbal  communications. 
Workload  level  did  not  have  a  significant  affect  on  this  variable.  There  were  no  significant  changes 
in  the  number  of  other  verbal  communications  from  the  Intelligence  Officer  to  the  Air  commander 
or  in  joint  communications  as  a  function  of  Shared  Display  Type  or  Workload  Level. 
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Figure  15.  Mean  Number  of  Requests  for  Aircraft  Identification  from  Air  Commander 
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Figure  16:  Number  of  Communications  from  the  Air  Commander  to  the  Intelligence  Officer 


Sensor  Performance 

The  third  analysis  examined  the  effect  of  the  sensor  performance  (all  sensors  reporting,  missing 
data,  dissonant  data),  sensor  pattern  (which  sensor  was  missing  or  dissonant),  and  sensor  reliability 
on  the  decision  making  of  the  Intelligence  Officer. 

Sensor  Pattern  Analysis 

During  the  study,  the  Intelligence  Officer  would  view  the  available  sensor  information  to  determine 
each  aircraft’s  classification.  Sensor  Performance  was  either  full  (all  three  sensors  reporting  the 
same  identification),  missing  (one  of  the  three  sensors  not  reporting),  or  dissonant  (one  of  the  three 
sensors  reporting  a  different  identification  from  the  other  two).  Four  Sensor  Patterns  were  presented 
for  missing  and  dissonant  data:  Sensor  A  missing  or  dissonant,  sensor  B  missing  or  dissonant, 
sensor  C  missing  or  dissonant,  and  sensors  B  and  C  missing  or  dissonant  from  sensor  A.  First,  an 
ANOVA  of  Sensor  Performance  by  Sensor  Pattern  was  performed  on  both  the  mean  viewing  time 
and  correctness  of  classification  by  the  Intelligence  Officer. 

Mean  viewing  time  was  significantly  effected  by  the  Sensor  Performance,  F(2,204)  =  12.824,  p  < 
.001,  but  not  Sensor  Pattern.  The  Intelligence  Officers  took  significantly  longer  to  make  a 
classification  decision  for  dissonant  data. 

The  correctness  of  the  classification  decision  was  significantly  effected  by  both  the  Sensor 
Performance  and  the  Sensor  Pattern.  Intelligence  Officers  incorrectly  classified  targets  with 
dissonant  sensor  data  more  frequently  than  they  did  those  with  full  or  missing  sensor  data,  F( 2,  204) 
=  167.674,  p  <  .001.  Surprisingly,  missing  sensor  data  was  not  a  significant  problem  for 
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classification  performance,  contrary  to  a  sensor  that  showed  dissonant  data.  It  should  be  noted  that 
a  missing  sensor  could  have  reported  either  the  same  or  dissonant  information,  therefore  it  had  the 
same  potential  for  dissonance.  The  Intelligence  Officers  did  not  appear  to  take  this  into  account, 
however.  The  pattern  of  the  dissonant  or  missing  sensor  also  had  an  effect,  F(3,204)  =  18.091,  p  < 
.001,  with  worse  performance  when  sensor  A  was  dissonant,  but  correct. 

The  sensor  performance  by  sensor  pattern  interaction  was  also  significant,  F(6,204)  =  13.02,  p  < 
.001.  The  interaction  appears  to  be  primarily  driven  by  the  dissonant  data,  as  shown  in  Figure  17. 
They  performed  the  worst  when  the  dissonant  data  was  presented  on  sensor  A,  the  first  in  the  list. 
Under  these  conditions,  the  Intelligence  Officer  correctly  identified  the  aircraft  less  than  20%  of  the 
time.  These  results  are  similar  to  that  found  in  the  earlier  study  (Bolstad  &  Endsley,  1998), 
however,  interesting  differences  are  present.  The  overall  classification  accuracy  was  worse  overall 
under  sensor  dissonance  and  the  degree  of  problem  experienced  when  sensor  A  was  dissonant  was 
greater. 

The  main  difference  between  the  two  studies  was  that  the  sensors  in  this  study  had  varying 
reliability  as  a  function  of  the  location  of  the  AWACS,  while  in  the  previous  study,  the  reliability 
remained  constant  for  each  sensor.  A  further  analysis  was  therefore  conducted  to  examine  the 
effects  of  the  sensor  reliabilities  on  the  strategies  used  by  the  Intelligence  Officers  for  dealing  with 
dissonant  data. 
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Figure  17.  Correctness  of  Classification  by  Sensor  Performance  and  Sensor  Pattern 
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Sensor  Reliabilities 


We  conducted  a  further  analysis  on  the  classification  strategies  utilized  by  the  Intelligence  Officers 
as  a  function  of  the  relative  reliabilities  of  the  sensors.  We  wished  to  determine  the  degree  to  which 
the  Intelligence  Officers  were  taking  sensor  reliability  into  account  versus  the  degree  to  which  they 
were  relying  on  other  heuristics,  such  as  reliance  on  the  first  sensor  in  the  list,  or  on  the 
identification  supported  by  most  sensors,  regardless  of  reliability. 

There  were  five  different  cases  of  dissonance  analyzed.  In  all  cases  of  information  dissonance,  at 
least  two  sensors  reported  the  same  identification  (majority)  and  the  other  sensor  was  different 
(minority).  The  five  possibilities  are  depicted  in  Table  2. 

Table  2.  Sensor  Reliability  Possibilities 


Majority 

Sensor  Data 
Aircraft  is  X 

Majority 

Sensor  Data 
Aircraft  is  X 

Minority 

Sensor  Data 
Aircraft  is  Y 

CASE  1 

1 

2 

2 

Rankings  of  Sensor  Reliabilities 

1 

2 

3 

1 

3 

2 

CASE  2 

Rankings  of  Sensor  Reliabilities 

2 

.  .  2 

1 

CASE  3 

Rankings  of  Sensor  Reliabilities 

1 

1 

■  2 

CASE  4 

Rankings  of  Sensor  Reliabilities 

. 1  .  v 

.  2 

1 

CASE  5 

Rankings  of  Sensor  Reliabilities 

1 

1 

1 

(Case  1)  One  of  the  majority  sensors  has  the  highest  reliability  and  the  other  two 
have  lower  reliabilities. 

(Case  2)  Both  majority  sensors  have  the  same  reliability,  but  the  minority  sensor  has 
the  highest  reliability. 

(Case  3)  Both  majority  sensors  have  the  same  reliability  and  it  is  higher  than  the 
reliability  of  the  minority  sensor. 

(Case  4)  The  minority  sensor  and  one  majority  sensor  have  the  highest  reliability  and  the 
other  majority  sensor  is  of  lower  reliability. 
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(Case  5)  All  sensors  have  the  same  reliability. 

The  Intelligence  Officer’s  classification  strategies  based  on  these  cases  is  shown  in  Figure  18.  In 
the  first  case,  one  of  the  majority  sensors  was  also  always  the  one  with  the  highest  reliability. 
Therefore,  these  two  strategies  are  not  discriminable  in  this  case.  They  accounted  for  the  majority, 
over  90%,  of  the  classifications.  This  same  pattern  dominated  Case  3,  in  which  both  majority 
sensors  had  the  same  reliability  and  it  was  higher  than  that  of  the  minority  sensor.  Very  few  other 
strategies  were  observed  in  these  cases. 

In  Case  4,  the  minority  sensor  and  only  one  of  the  majority  sensors  had  the  highest  reliability  (e.g. 
F/A-18-75%,  F16-75%,  F16-50%).  The  other  majority  sensor  was  of  lower  reliability.  In  this  case, 
both  choices  were  of  the  highest  reliability.  In  examining  whether  participants  chose  the  minority 
or  majority  sensor  recommendation  in  this  case,  it  appears  that  they  were  far  more  likely  to  choose 
the  identification  of  the  majority  sensor  (68%),  than  that  of  the  minority  sensor  (28%),  although  not 
an  insignificant  number  chose  the  identification  of  minority  sensor. 

Case  2  allowed  discrimination  between  the  strategies  of  choosing  the  majority  or  the  most  reliable 
sensor.  In  this  case,  the  minority  sensor  had  the  highest  reliability  (e.g.  FI 6-50%,  F/A- 18-25%, 
F/A- 18-25%).  Decision  behavior  appeared  close  to  chance  in  this  ambiguous  case,  with  the 
likelihood  of  selecting  the  identification  of  the  more  highly  reliable  minority  sensor  (49%)  roughly 
equaling  the  likelihood  of  selecting  the  identification  of  the  less  reliable  majority  sensors  (47.5%). 

Case  5,  in  which  the  sensors  all  had  equal  reliability,  also  created  difficulties  for  decision  making. 
While  the  most  frequent  strategy  observed  in  this  case  was  to  adopt  the  recommendation  of  the 
majority  sensor  (79%),  many  people  also  followed  the  minority  sensor  (18%). 
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Figure  18.  Strategies  Used  in  Dealing  with  Information  Dissonance 


Overall,  it  appears  that  participants  did  take  sensor  reliability  as  well  as  sensor  confirmation  into 
account  in  their  decision  processes.  In  comparison,  a  simpler  strategy  of  picking  the 
recommendation  of  the  first  sensor  in  the  list  (A)  was  not  supported  by  the  data.  In  Cases  1  and  5, 
selection  of  the  recommendation  of  the  first  sensor  was  at  chance  levels  and  also  indistinguishable 
from  that  expected  by  the  strategy  of  selecting  the  majority  sensor  (of  the  two  choices,  there  was  a 
50%  chance  of  the  majority  sensor  choice  being  in  place  A).  In  Case  2  and  Case  4,  participants 
were  significantly  less  likely  than  chance  to  have  picked  the  recommendations  of  the  first  sensor 
(pc.OOl).  In  these  cases  when  the  first  sensor  was  not  the  dissonant  sensor,  participants  were  less 
likely  than  would  be  expected  to  have  selected  its  recommendations.  Thus,  sensor  reliability  and 
sensor  confirmation  rather  than  sensor  position  appeared  to  dominate  the  strategies  used.  In  Case  3, 
the  opposite  was  found.  Participants  were  more  likely  than  expected  to  have  picked  the 
recommendations  of  the  first  sensor,  which  is  odd  considering  that  the  first  sensor  was  dissonant 
more  of  the  time  in  these  cases  (pc.OOl).  Sensor  reliability  appeared  to  be  a  more  important 
consideration  in  dealing  with  dissonance. 

Finally,  we  examined  the  degree  to  which  the  Workload  Level  or  Shared  Display  Type  may  have 
affected  strategy  selection  in  this  task.  Shared  Display  Type  had  a  significant  effect  on  the 
strategies  used.  An  examination  of  Figure  19  shows  that  while  the  trends  are  the  same  across 
conditions,  there  was  more  of  a  tendency  to  rely  on  high  reliability  sensors  in  the  Non-Shared 
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Display  condition  (jfi  =  10.7,  p  <  .01),  and  an  increased  tendency  to  rely  on  the  majority  sensors  in 
the  Full  Shared  and  Abstracted  Shared  Display  conditions  (^2  =  7.09,  p  <  .05).  They  were  also 
more  likely  to  use  other  strategies  (or  not  classify  targets)  under  the  Full  Shared  Displays  Condition 
(X2  =  16.7,/?  <  .001). 


Non-Shared  Displays  Full  Shared  Displays  Abstracted  Shared 

Displays 


Figure  19.  Strategy  by  Shared  Display  Type 


Workload  level  also  significantly  affected  identification  strategy,  Figure  20.  There  was  less  of  a 
tendency  to  rely  on  the  more  reliable  sensors  under  high  workload  {%2  =  12.01,  p  <  .01),  and  a 
greater  tendency  to  use  the  first  sensor  under  low  workload  (%2  =  8.48,  p  <  .05).  They  appeared  to 
use  other  strategies  (or  not  classify  targets)  more  often  under  high  workload  (j2  =  23.2,  p  <  .001). 
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Figure  20.  Strategy  by  Workload  Level 


DISCUSSION 

This  study  indicates  that  the  way  in  which  people  use  Shared  Displays  is  actually  quite  complex  and 
related  to  both  task  strategies  used  and  workload  level.  As  compared  to  a  situation  in  which  no 
shared  display  is  available,  providing  Full  Shared  Displays  induced  lower  reward  and  higher 
penalty  points  overall.  Examination  of  the  data  shows  that  these  displays  were  problematic 
particularly  under  high  workload  levels.  The  extra  information  served  to  hinder  performance  under 
high  workload. 

The  Abstracted  Shared  Displays  provided  more  of  the  benefits,  with  less  of  the  problems  of  the  Full 
Shared  Displays.  The  penalty  points  were  lower  than  with  Shared  Displays,  particularly  the 
moderate  and  high  workload  levels. 
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Under  low  workload  levels,  no  performance  enhancements  or  problems  were  found  associated  with 
either  of  the  Shared  Display  types.  Under  these  conditions,  participants  were  able  to  achieve  the 
needed  team  SA  using  purely  verbal  communications.  It  is  expected  that  as  task  complexity 
increases,  doing  so  will  be  harder  and  harder,  however.  As  Workload  Level  increased,  the 
Abstracted  Shared  Displays  proved  to  be  a  particular  benefit,  outperforming  both  the  Non-Shared 
and  Full  Shared  Display  conditions. 

Interestingly,  the  means  by  which  this  occurred  was  not  so  readily  discemable.  An  increase  in  the 
degree  to  which  the  Intelligence  Officers  were  following  the  Air  Commander’s  prioritization  list 
was  not  evident  and  overall  verbal  communications  from  the  Air  Commander  to  the  Intelligence 
Officer  decreased  in  the  Abstracted  Shared  Display  condition,  compared  to  the  other  display 
conditions.  Other  verbal  communications  remained  unchanged  between  the  display  conditions.  It 
can  only  be  speculated  that  these  two  indices  of  overt  team  communication  were  rendered 
unnecessary  by  the  improved  information  sharing  which  was  possible  with  the  Abstracted  Shared 
Display. 

The  use  of  an  abstracted  display  also  seems  to  decrease  the  need  for  team  interaction.  Teams  using 
abstracted  displays  requested  fewer  targets  and  had  overall  lower  level  of  verbal  exchanges.  These 
teams  may  have  had  enough  sufficient  information  to  perform  their  task  and  only  request 
information  from  the  Intelligence  Officer  when  they  need  verification.  On  the  other  hand,  teams 
using  verbal  displays  never  developed  a  mental  model  of  the  other  persons  task  and  therefore  did 
not  fully  understand  how  the  team  functions.  These  teams  had  fewer  requests  for  target 
classifications  and  a  greater  number  of  verbal  exchanges.  However,  they  may  not  have  fully 
understood  what  information  their  team  member  needed  and  were  passing  irrelevant  information. 

By  varying  workload  level  we  hoped  to  further  understand  how  the  strategies  used  by  teams  change 
as  the  workload  level  increases  (from  three  aircraft  to  nine  aircraft)  and  create  more  team 
interaction.  What  we  found  was  a  greater  increase  in  both  penalty  and  reward  points  as  the  task  got 
more  difficult,  which  was  true  for  all  the  display  types.  Teams  had  the  greatest  opportunity  to 
accrue  points  as  the  workload  level  increased.  They  also  made  the  largest  number  of  errors  and 
overall  performance  decreased.  There  was  also  less  team  interaction  as  workload  increased. 

Thus,  it  can  be  said  that  this  study  serves  to  further  confirm  certain  aspects  of  the  team  SA  model. 
Namely,  team  SA  can  be  supported  through  shared  displays  and  one  can  compensate  for  the  other  in 
achieving  the  team  SA  needed  to  support  joint  task  performance.  The  use  of  Abstracted  Shared 
Displays  over  Full  Shared  Displays  is  supported  by  this  research,  particularly  under  high  workload 
levels. 

Secondly,  the  performance  of  participants  in  milking  decisions  in  the  face  of  information  dissonance 
was  further  explored.  Like  the  previous  study,  we  found  that  dissonant  data  is  handled  very 
differently  than  missing  data.  The  Intelligence  Officer  took  significantly  longer  to  make 
classifications  with  missing  or  dissonant  data.  We  also  wanted  to  determine  if  participants  were 
relying  on  the  sensor  reliabilities  to  determine  classification  when  data  is  dissonant.  It  was  found 
that  with  varying  sensor  reliabilities,  participants  relied  upon  strategies  that  took  account  of  both  the 
reliability  of  the  sensors  and  the  presence  of  confirming  sensors  in  making  their  decisions. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


In  conclusion,  this  study  supported  the  use  of  shared  displays  for  enhancing  team  performance.  The 
design  of  these  displays  needs  to  be  carefully  constructed,  however,  to  provide  each  team  member 
with  just  the  pertinent  information  required  to  maintain  team  SA.  That  is,  it  needs  to  be  constructed 
based  upon  an  analysis  of  the  team  members  shared  SA  requirements.  This  finding  supports  the 
first  component  of  the  team  SA  model. 

Secondly,  it  was  found  the  degree  to  which  team  members  will  function  as  a  team  and  will  benefit 
from  shared  displays  is  dependant  on  their  workload  level.  At  low  levels  of  workload,  team 
processes  and  verbal  communications  may  be  dependent  for  forming  the  needed  team  SA.  At 
moderate  and  higher  workload  levels,  teams  will  be  negatively  affected  by  shared  displays  that 
provide  too  much  information,  but  positively  affected  by  shared  displays  that  provide  just  that 
information  needed  for  shared  SA. 

Finally,  this  study  shed  additional  light  on  the  handling  of  information  dissonance  in  command  and 
control  type  tasks.  Specifically,  dissonant  information  was  found  to  both  slow  performance  and 
decrease  performance  accuracy.  Participants  were  sensitive  to  both  sensor  reliability  and  sensor 
confirmation  in  making  their  decisions;  however,  the  ambiguity  present  in  these  situations  was  hard 
to  overcome  with  their  decision  strategies. 

Overall,  this  study  confirmed  and  expanded  on  the  previous  research  on  team  SA.  Team  SA  and 
performance  in  team  tasks  can  be  greatly  enhanced  by  the  development  of  shared  displays  that  are 
based  on  the  shared  information  requirements  of  team  members.  More  research  is  needed  to  find 
ways  to  combat  the  observed  problems  in  handing  information  dissonance.  In  addition,  more 
research  is  needed  to  expand  these  results  to  more  realistic  tasks  and  battlefield  conditions. 
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APPENDIX  A  :  Intelligence  Officer  Job  Description  for  Non-Shared  Display  and  Full 
Shared  Displays 


Overview 

In  this  study  we  are  interested  in  understanding  how  individual  and  team  decision  making  occurs  in 
the  warfare  environment.  Unlike  the  wars  fought  in  the  earlier  part  of  this  century,  future  battles 
will  have  individuals  working  together  that  are  separated  by  great  distances,  such  as  continents. 
Thus,  the  transfer  and  understanding  of  information  between  teams  will  be  critical  to  mission 
success. 

During  this  task  you  will  be  asked  to  make  decisions  that  are  similar  to  ones  faced  by  individuals 
during  a  military  operation.  You  will  be  assigned  to  either  the  position  of  air  commander  or 
intelligence  officer  (Intel  officer).  Your  battle  ground  is  the  air  space  over  your  home  base.  You 
will  be  working  together  to  defeat  the  enemy.  It  is  the  air  commander’s  job  to  ensure  that  the  home 
base  is  protected  from  enemy  aircraft.  The  air  commander  will  be  viewing  these  aircraft  on  a 
display  similar  to  a  radar  screen.  The  Intel  officer’s  job  entails  providing  the  air  commander  with 
information  on  these  same  aircraft  that  are  detected  by  the  sensors.  Your  success  will  depend  upon 
how  well  the  overall  team  performs. 

This  task  is  hosted  on  two  separate  computer  workstations.  You  will  be  asked  to  sit  at  one  of  these 
workstations  while  wearing  a  set  of  headphones.  We  will  first  begin  with  a  training  period  followed 
by  3  trials  of  approximately  6  minutes  each.  You  may  request  a  break  at  any  time  between  trials 
and  you  will  be  given  a  15  minute  break  during  the  middle  of  the  test.  This  will  be  followed  by  a 
second  test  period  of  6  more  trials.  If  you  have  any  questions  regarding  the  following  task 
instructions  please  ask  them  before  the  testing  begins.  You  will  get  to  practice  before  testing  begins 
and  may  also  ask  questions  during  practice. 

Intelligence  Officer 

The  United  States  has  just  learned  that  Ikestan  attacked  a  U.S.  base  in  the  middle  east.  The 
president  has  ordered  all  military  personnel  in  the  surrounding  areas  to  be  on  alert  for  possible 
attacks.  You  are  stationed  at  an  air  base  in  Western  Europe.  A  few  hours  after  the  initial  attack  you 
are  instructed  to  report  to  your  post  as  enemy  aircraft  are  making  their  way  towards  a  U.S.  base  in 
N.  Africa.  Aircraft  have  been  sent  to  intercept  and  destroy  these  incoming  planes.  It  is  your  job  to 
provide  the  air  commander  at  this  N.  African  base  with  target  identification  information  on  these 
incoming  aircraft. 

You  will  be  providing  requested  target  information.  You  will  be  shown  a  list  of  aircraft  target 
numbers  that  are  within  this  base’s  airspace.  It  is  your  job  to  decide  on  target  identifications  and  to 
send  this  information  to  the  air  commander. 

You  may  see  6  different  types  of  aircraft:  friendly  fighters,  enemy  fighters,  friendly  bombers, 
enemy  bombers,  friendly  transports,  and  enemy  transports.  There  are  a  total  of  16  different  kinds  of 
aircraft  that  are  in  use  by  friendly  and  enemy  forces  as  shown  in  the  table  below. 


31 


Plane  Categories 
Friendly  Fighters 

Types 

F/A-18 

F-15E 

F-16 

Friendly  Bombers 

B-52 

B-l 

B-2 

Friendly  Transports 

Enemy  Fighters 

C-130J 

C-21 

KC-135 

Mig-29 

Su-35 

Su-37 

Enemy  Bombers 

Tu-22M 

Tu-168 

Enemy  Transports 

An- 124 
An-225 

Your  station  contains  a  set  of  head  phones  and  the  information  screen  that  is  shown  on  the 
following  page.  Targets  whose  ID  are  requested  are  shown  in  the  box  at  the  top  of  the  screen 
labeled  ‘Requested  Information’  in  order  of  priority.  You  need  to  monitor  this  section  of  your 
screen  as  the  target  numbers  will  be  continually  changing  as  targets  land,  are  destroyed,  or  appear  in 
your  air  space. 

You  will  have  access  to  sensor  data  about  these  targets.  The  screen  will  display  each  sensors  best 
assessment  of  the  target’s  ID,  which  may  not  always  be  correct.  It  is  up  to  you  to  determine  the  true 
aircraft  identification,  so  use  your  best  judgment.  You  can  access  sensor  data  for  a  target  by 
clicking  on  the  ‘View  Information’  button  that  corresponds  to  the  target  number  or  by  pressing  a  the 
number  on  the  keyboard  that  corresponds  to  the  button  number  on  the  screen.  However,  you  may 
only  see  sensor  data  for  one  target  at  a  time. 

The  sensors  are  labeled  ‘Source  A’,  ‘Source  B’,  and  ‘Source  C’.  AW  ACS,  also  known  as 
reconnaissance  planes,  are  providing  you  with  this  information.  The  reliability  of  the  sensor 
information  is  dependent  upon  their  location,  the  closer  the  plane  to  home  base  (center  of  the  grid), 
the  more  reliable  the  information.  Remember  to  use  these  reliabilities  when  determining  target 
identification.  Be  aware  that  not  all  sensors  may  report  information  on  a  particular  target.  For 
instance,  Sensors  B  and  C  may  have  information  on  the  first  target  requested,  but  not  sensor  A. 
Thus,  you  may  have  to  make  your  decision  on  partial  information. 

Once  you  have  viewed  the  sensor  data,  you  need  to  indicate  the  aircraft’s  identification  by  selecting 
the  aircraft  type  on  the  right  side  of  your  screen.  All  the  possible  aircraft  are  listed  there.  After  you 
have  selected  an  ID,  click  on  the  ‘Send  Information’  button  to  send  this  information  to  the  air 
commander. 
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The  bottom  part  of  the  screen  will  provide  you  with  information  regarding  the  outcome  of  the 
battle.  It  will  display  the  aircraft  number  and  its  actual  ID  as  well  as  the  final  result  for  the  plane; 
destroyed,  got  through,  or  collided.  This  feedback  will  give  you  information  regarding  your 
classification  of  the  aircraft.  If  you  misclassified  a  plane,  the  “Classified  As”  and  “Actual  Type” 
box  will  display  two  different  airplanes  and  the  text  will  appear  in  red.  If  you  correctly  classified  an 
aircraft,  the  text  will  appear  in  blue.  If  the  plane  was  not  classified  prior  to  resolution,  it  will  appear 
in  green  text.  Use  this  information  to  help  you  better  identify  the  aircraft.  There  will  be  a  delay 
between  making  your  ID  and  receiving  this  feedback,  as  true  ID  and  disposition  cannot  be 
determined  until  the  plane  has  either  landed  or  been  destroyed. 

Sensor  data  for  aircraft  will  continually  appear  on  your  information  screen  until  the  testing  time  is 
up.  Thus,  you  could  potentially  process  hundreds  of  planes  in  the  6  minutes  allotted  for  the  task. 
Please  be  aware  that  this  task  moves  quickly.  We  want  you  to  work  as  quickly  and  as  accurately  as 
you  can. 
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APPENDIX  B  :  Intelligence  Officer  Job  Description  for  Abstract  Shared  Displays 


Overview 

In  this  study  we  are  interested  in  understanding  how  individual  and  team  decision  making  occurs  in 
the  warfare  environment.  Unlike  the  wars  fought  in  the  earlier  part  of  this  century,  future  battles 
will  have  individuals  working  together  that  are  separated  by  great  distances,  such  as  continents. 
Thus,  the  transfer  and  understanding  of  information  between  teams  will  be  critical  to  mission 
success. 

During  this  task  you  will  be  asked  to  make  decisions  that  are  similar  to  ones  faced  by  individuals 
during  a  military  operation.  You  will  be  assigned  to  either  the  position  of  air  commander  or 
intelligence  officer  (Intel  officer).  Your  battle  ground  is  the  air  space  over  your  home  base.  You 
will  be  working  together  to  defeat  the  enemy.  It  is  the  air  commander’s  job  to  ensure  that  the  home 
base  is  protected  from  enemy  aircraft.  The  air  commander  will  be  viewing  these  aircraft  on  a 
display  similar  to  a  radar  screen.  The  Intel  officer’s  job  entails  providing  the  air  commander  with 
information  on  these  same  aircraft  that  are  detected  by  the  sensors.  Your  success  will  depend  upon 
how  well  the  overall  team  performs. 

This  task  is  hosted  on  two  separate  computer  workstations.  You  will  be  asked  to  sit  at  one  of  these 
workstations  while  wearing  a  set  of  headphones.  We  will  first  begin  with  a  training  period  followed 
by  3  trials  of  approximately  6  minutes  each.  You  may  request  a  break  at  any  time  between  trials 
and  you  will  be  given  a  15  minute  break  during  the  middle  of  the  test.  This  will  be  followed  by  a 
second  test  period  of  6  more  trials.  If  you  have  any  questions  regarding  the  following  task 
instructions  please  ask  them  before  the  testing  begins.  You  will  get  to  practice  before  testing  begins 
and  may  also  ask  questions  during  practice. 

Intelligence  Officer 

The  United  States  has  just  learned  that  Ikestan  attacked  a  U.S.  base  in  the  middle  east.  The 
president  has  ordered  all  military  personnel  in  the  surrounding  areas  to  be  on  alert  for  possible 
attacks.  You  are  stationed  at  an  air  base  in  Western  Europe.  A  few  hours  after  the  initial  attack  you 
are  instructed  to  report  to  your  post  as  enemy  aircraft  are  making  their  way  towards  a  U.S.  base  in 
N.  Africa.  Aircraft  have  been  sent  to  intercept  and  destroy  these  incoming  planes.  It  is  your  job  to 
provide  the  air  commander  at  this  N.  African  base  with  target  identification  information  on  these 
incoming  aircraft. 

You  will  be  providing  requested  target  information.  You  will  be  shown  a  list  of  aircraft  target 
numbers  that  are  within  this  base’s  airspace.  It  is  your  job  to  decide  on  target  identifications  and  to 
send  this  information  to  the  air  commander. 

You  may  see  6  different  types  of  aircraft:  friendly  fighters,  enemy  fighters,  friendly  bombers, 
enemy  bombers,  friendly  transports,  and  enemy  transports.  There  are  a  total  of  16  different  kinds  of 
aircraft  that  are  in  use  by  friendly  and  enemy  forces  as  shown  in  the  table  below. 
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Plane  Categories 
Friendly  Fighters 

Types 

F/A-18 

F-15E 

F-16 

Friendly  Bombers 

B-52 

B-l 

B-2 

Friendly  Transports 

C-130J 

C-21 

KC-135 

Enemy  Fighters 

Mig-29 

Su-35 

Su-37 

Enemy  Bombers 

Tu-22M 

Tu-168 

Enemy  Transports 

An- 124 
An-225 

Your  station  contains  a  set  of  head  phones  and  the  information  screen  that  is  shown  on  the 
following  page.  Targets  whose  ID  are  requested  are  shown  in  the  box  at  the  top  of  the  screen 
labeled  ‘Requested  Information’  in  order  of  priority.  You  need  to  monitor  this  section  of  your 
screen  as  the  target  numbers  will  be  continually  changing  as  targets  land,  are  destroyed,  or  appear  in 
your  air  space. 

Target  location  is  also  provided  in  the  lower  right  part  of  your  screen.  The  target  numbers  are 
ordered  according  to  their  proximity  to  the  home  base.  The  closest  target  is  at  the  top  of  the  list  and 
the  farthest  is  at  the  bottom  of  the  list. 

You  will  have  access  to  sensor  data  about  these  targets.  The  screen  will  display  each  sensors  best 
assessment  of  the  target’s  ID,  which  may  not  always  be  correct.  It  is  up  to  you  to  determine  the  true 
aircraft  identification,  so  use  your  best  judgment.  You  can  access  sensor  data  for  a  target  by 
clicking  on  the  ‘View  Information’  button  that  corresponds  to  the  target  number  or  by  pressing  the 
number  on  the  keyboard  that  corresponds  to  the  button  number  on  the  screen.  However,  you  may 
only  see  sensor  data  for  one  target  at  a  time. 

The  sensors  are  labeled  ‘Source  A’,  ‘Source  B’,  and  ‘Source  C’.  AWACs,  also  known  as 
reconnaissance  planes,  are  providing  you  with  this  information.  The  reliability  of  the  sensor 
information  is  dependent  upon  their  location,  the  closer  the  plane  to  home  base  (center  of  the  grid), 
the  more  reliable  the  information.  The  reliabilities  of  the  sensors  is  shown  in  the  lower  right  comer 
or  your  screen.  The  reliabilities  will  either  be  25%,  50%,  or  75%.  Remember  to  use  these 
reliabilities  when  determining  target  identification.  Be  aware  that  not  all  sensors  may  report 
information  on  a  particular  target.  For  instance.  Sensors  B  and  C  may  have  information  on  the  first 
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target  requested,  but  not  sensor  A.  Thus,  you  may  have  to  make  your  decision  on  partial 
information. 


Once  you  have  viewed  the  sensor  data,  you  need  to  indicate  the  aircraft’s  identification  by  selecting 
the  aircraft  type  on  the  right  side  of  your  screen.  All  the  possible  aircraft  are  listed  there.  After  you 
have  selected  an  ID,  click  on  the  ‘Send  Information’  button  to  send  this  information  to  the  air 
commander. 

The  bottom  part  of  the  screen  will  provide  you  with  information  regarding  the  outcome  of  the 
battle.  It  will  display  the  aircraft  number  and  its  actual  ID  as  well  as  the  final  result  for  the  plane; 
destroyed,  got  through,  or  collided.  This  feedback  will  give  you  information  regarding  your 
classification  of  the  aircraft.  If  you  misclassified  a  plane,  the  “Classified  As”  and  “Actual  Type” 
box  will  display  two  different  airplanes  and  the  text  will  appear  in  red.  If  you  correctly  classified  an 
aircraft,  the  text  will  appear  in  blue.  If  the  plane  was  not  classified  prior  to  resolution,  it  will  appear 
in  green  text.  Use  this  information  to  help  you  better  identify  the  aircraft.  There  will  be  a  delay 
between  making  your  ID  and  receiving  this  feedback,  as  true  ID  and  disposition  cannot  be 
determined  until  the  plane  has  either  landed  or  been  destroyed. 

Your  overall  team  success  is  determined  by  the  number  of  reward  and  penalty  points  you  accrue. 
Point  assignments  have  been  made  for  each  type  of  aircraft  representing  the  reward  points  for 
destroying  the  aircraft  and  the  penalty  points  for  allowing  the  aircraft  to  land  at  your  home  base. 
This  information  is  provided  in  the  lower  right  comer  of  your  screen. 

Sensor  data  for  aircraft  will  continually  appear  on  your  information  screen  until  the  testing  time  is 
up.  Thus,  you  could  potentially  process  hundred  of  planes  in  the  6  minutes  allotted  for  the  task. 
Please  be  aware  that  this  task  moves  quickly.  We  want  you  to  work  as  quickly  and  as  accurately  as 
you  can. 
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APPENDIX  C  :  Air  Commander  Job  Description  for  Non-Shared  Display  and  Full 
Shared  Displays 


Overview 

In  this  study  we  are  interested  in  understanding  how  individual  and  team  decision  making  occurs  in 
the  warfare  environment.  Unlike  the  wars  fought  in  the  earlier  part  of  this  century,  future  battles 
will  have  individuals  working  together  that  are  separated  by  great  distances,  such  as  continents. 
Thus,  the  transfer  and  understanding  of  information  between  teams  will  be  critical  to  mission 
success. 

During  this  task  you  will  be  asked  to  make  decisions  that  are  similar  to  ones  faced  by  individuals 
during  a  military  operation.  You  will  be  assigned  to  either  the  position  of  air  commander  or 
intelligence  officer  (Intel  officer).  Your  battle  ground  is  the  air  space  over  your  home  base.  You 
will  be  working  together  to  defeat  the  enemy.  It  is  the  air  commander’s  job  to  ensure  that  the  home 
base  is  protected  from  enemy  aircraft.  The  air  commander  will  be  viewing  these  aircraft  on  a 
display  similar  to  a  radar  screen.  The  Intel  officers  job  entails  providing  the  air  commander  with 
information  on  these  same  aircraft  that  are  detected  by  the  sensors.  Your  success  will  depend  upon 
how  well  the  overall  team  performs. 

This  task  is  hosted  on  two  separate  computer  workstations.  You  will  be  asked  to  sit  at  one  of  these 
workstations  while  wearing  a  set  of  headphones.  We  will  first  begin  with  a  training  period  followed 
by  3  trials  of  approximately  6  minutes  each.  You  may  request  a  break  at  any  time  between  trials 
and  you  will  be  given  a  15  minute  break  during  the  middle  of  the  test.  This  will  be  followed  by  a 
second  test  period  of  6  more  trials.  If  you  have  any  questions  regarding  the  following  task 
instructions  please  ask  them  before  the  testing  begins.  You  will  get  to  practice  before  testing  begins 
and  may  also  ask  questions  during  practice. 

Air  Commander 

The  United  States  has  just  learned  that  Ikestan  attacked  a  U.S.  base  in  the  middle  east.  The 
president  has  ordered  all  military  personnel  in  the  surrounding  areas  to  be  on  alert  for  possible 
attacks.  You  are  stationed  at  an  air  base  in  Tenya,  N.  Africa,  which  has  been  put  on  alert.  A  few 
hours  after  the  initial  attack  you  are  instructed  to  report  to  your  post  as  enemy  aircraft  are  making 
their  way  towards  your  base.  Aircraft  have  been  sent  to  intercept  and  destroy  these  incoming 
planes.  It  is  your  job  to  protect  your  base  from  enemy  aircraft  strikes  while  allowing  your  planes  to 
land  safely. 

Your  success  is  determined  by  the  number  of  reward  and  penalty  points  you  accrue.  You  will  want 
to  maximize  your  reward  points  and  minimize  your  penalty  points.  In  order  to  do  this,  you  will 
need  to  determine  which  aircraft  are  friendly  and  which  are  enemy  and  what  type  of  aircraft  they 
are.  At  the  same  time,  you  will  need  to  prioritize  incoming  targets  based  upon  their  range  from 
your  home  base  and  speed  to  determine  which  aircraft  are  the  most  critical  at  any  one  time.  You 
will  request  aircraft  identity  information  from  the  Intel  officer  (who  is  at  an  airbase  in  Western 
Europe)  to  support  this  prioritization  decision. 
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Point  assignments  have  been  made  for  each  type  of  aircraft  representing  the  reward  points  for 
destroying  the  aircraft  and  the  penalty  points  for  allowing  the  aircraft  to  land  at  your  home  base. 

The  points  are  based  on  the  mission  relevance  and  lethality  of  the  aircraft.  For  example,  transports 
can  carry  many  personnel  and  therefore  could  present  a  great  threat  if  they  land  at  a  U.S.  base,  while 
the  loss  of  a  U.S.  transport  would  be  a  devastating  blow  to  our  forces.  Friendly  aircraft  have  a  zero 
penalty  for  getting  through  to  home  base  (you  would  want  them  to  safely  land  at  home)  and  a 
negative  reward  associated  with  destroying  it  (you  would  not  want  to  destroy  your  own  planes). 
Enemy  aircraft,  however,  have  both  positive  reward  and  penalty  points. 

In  addition,  it  is  possible  for  some  targets  to  collide  with  one  another.  If  two  friendly  aircraft 
collide  no  points  are  accrued  for  this.  If  two  enemy  aircraft  collide,  the  highest  reward  points  of  the 
two  aircraft  are  given.  However,  if  a  friendly  and  an  enemy  aircraft  collide,  you  receive  double  the 
penalty  points  of  the  enemy  aircraft.  Therefore,  it  is  in  your  best  interest  to  destroy  enemy  aircraft 
before  they  collide  with  friendly  planes. 

You  will  see  6  different  types  of  aircraft:  friendly  fighters,  enemy  fighters,  friendly  bombers,  enemy 
bombers,  friendly  transports,  and  enemy  transports.  There  are  a  total  of  16  different  kinds  of 
aircraft  that  are  in  use  by  friendly  and  enemy  forces.  Below  is  a  table  of  these  aircraft,  their  color 
appearance  on  your  radar  screen  once  identified,  and  their  associated  reward  and  penalty  points. 


Plane  Categories 

Friendly  Fighters 

Types 

F/A-18 

F-15E 

F-16 

Color 

Blue 

Blue 

Blue 

Reward 

Points 

-20 

-40 

-60 

Penalty  Points 

0 

0 

0 

Friendly  Bombers 

B-52 

Green 

-50 

0 

B-l 

Green 

-80 

0 

B-2 

Green 

-100 

0 

Friendly  Transports 

C-130J 

Turquoise 

-120 

0 

C-21 

Turquoise 

-140 

0 

KC-135 

Turquoise 

-150 

0 

Enemy  Fighters 

Mig-29 

Red 

60 

10 

Su-35 

Red 

80 

20 

Su-37 

Red 

100 

10 

Enemy  Bombers 

Tu-22M 

Orange 

10 

50 

Tu-168 

Orange 

20 

60 

Enemy  Transports 

An- 124 

Yellow 

50 

50 

An-225 

Yellow 

60 

60 

You  will  view  the  aircraft  on  a  radar  screen,  similar  to  that  shown  on  the  following  page.  Your 
home  base  is  at  the  center  of  the  screen.  The  aircraft  will  appear  from  outside  the  radar  and  travel 
inward  towards  your  home  base.  It  is  your  job  to  let  the  friendly  aircraft  land  at  the  base,  but  the 
enemy  aircraft  must  be  destroyed  before  reaching  the  base.  An  aircraft  will  get  through  if  left  alone 
(it  will  land  on  its  own). 
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You  destroy  aircraft  by  launching  missiles  at  them.  There  are  two  types  of  missiles  available;  a 
Sparrow  which  is  smaller  and  faster  and  used  for  destroying  fighters  and  AMRAAM  (advanced 
medium  range  air  to  air  missiles)  which  are  long  range  radar  guided  missiles  used  for  destroying 
bombers  and  transports.  To  launch  missiles  at  an  aircraft,  use  the  computer  mouse  to  click  on  it 
once;  the  left  mouse  button  launches  Sparrows  and  the  right  mouse  button  launches  AMRAAMs. 
This  gives  the  command  to  launch  missile  defense  resources  against  the  target.  The  target  will  turn 
a  light  violet  color  to  indicate  that  it  has  been  targeted.  If  you  launch  an  inappropriate  missile  at  an 
aircraft,  the  plane  may  not  be  destroyed  and  you  will  have  to  try  again. 

AWACS,  also  known  as  reconnaissance  planes,  will  also  be  present  on  your  screen.  They  are  small 
blue  circles  labeled  A,  B,  or  C.  It  is  their  job  to  determine  plane  identifications  and  relay  this 
information  to  the  Intel  Officer.  The  reliability  of  the  AW  AC  information  is  dependent  upon  their 
location,  the  closer  the  plane  to  home  base  (center  of  the  grid),  the  more  reliable  the  information. 

At  5  to  10  miles  out  (first  two  rings)  the  reliability  is  75%  at  15  to  20  miles  the  reliability  is  50% 
and  at  25  to  30  miles  the  reliability  is  25%. 

Initially,  all  aircraft  appear  as  white  squares.  Below  each  square  is  the  aircraft  target  number  and  its 
speed.  In  order  for  you  to  determine  what  type  of  aircraft  is  on  your  radar,  you  must  request  this 
information  from  the  intelligence  officer.  This  is  done  by  typing  the  aircraft’s  number  in  the  request 
box  in  the  lower  left  comer  of  the  radar  screen.  All  you  need  to  do  is  type  in  each  number  followed 
by  the  return  key.  The  Intel  officer  will  receive  and  process  your  information.  It’s  identity  will  be 
shown  by  the  color  of  the  square  on  your  radar  screen  (ex.  blue  for  friendly  fighters).  The  aircraft 
number  will  remain  the  same,  but  the  penalty  points  for  the  identification  that  has  been  made  will 
appear  below  the  target  number.  This  may  or  may  not  be  correct  depending  on  the  accuracy  of  the 
ID.  You  must  decide  what  to  do  with  each  aircraft,  either  processing  it  (destroying  it )  or  letting  it 
land  at  the  base.  At  all  times  during  this  task,  your  total  reward  and  penalty  points  are  shown  at  the 
top  left  comer  of  your  radar  screen. 

Aircraft  will  continually  appear  on  your  radar  screen  until  the  testing  time  is  up.  Thus,  you  could 
potentially  process  hundreds  of  planes  in  the  6  minutes  allotted  for  each  session.  Please  be  aware 
that  this  task  moves  quickly.  We  want  you  to  work  as  quickly  and  as  accurately  as  you  can.  These 
instructions  may  seem  a  bit  confusing,  so  we  will  start  with  several  training  sessions  to  help  clarify 
your  task. 
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APPENDIX  D  :  Air  Commander  Job  Description  for  Abstract  Shared  Displays 

Overview 

In  this  study  we  are  interested  in  understanding  how  individual  and  team  decision  making  occurs  in 
the  warfare  environment.  Unlike  the  wars  fought  in  the  earlier  part  of  this  century,  future  battles 
will  have  individuals  working  together  that  are  separated  by  great  distances,  such  as  continents. 
Thus,  the  transfer  and  understanding  of  information  between  teams  will  be  critical  to  mission 
success. 

During  this  task  you  will  be  asked  to  make  decisions  that  are  similar  to  ones  faced  by  individuals 
during  a  military  operation.  You  will  be  assigned  to  either  the  position  of  air  commander  or 
intelligence  officer  (Intel  officer).  Your  battle  ground  is  the  air  space  over  your  home  base.  You 
will  be  working  together  to  defeat  the  enemy.  It  is  the  air  commander’s  job  to  ensure  that  the  home 
base  is  protected  from  enemy  aircraft.  The  air  commander  will  be  viewing  these  aircraft  on  a 
display  similar  to  a  radar  screen.  The  Intel  officer’s  job  entails  providing  the  air  commander  with 
information  on  these  same  aircraft  that  are  detected  by  the  sensors.  Your  success  will  depend  upon 
how  well  the  overall  team  performs. 

This  task  is  hosted  on  two  separate  computer  workstations.  You  will  be  asked  to  sit  at  one  of  these 
workstations  while  wearing  a  set  of  headphones.  We  will  first  begin  with  a  training  period  followed 
by  3  trials  of  approximately  6  minutes  each.  You  may  request  a  break  at  any  time  between  trials 
and  you  will  be  given  a  15  minute  break  during  the  middle  of  the  test.  This  will  be  followed  by  a 
second  test  period  of  6  more  trials.  If  you  have  any  questions  regarding  the  following  task 
instructions  please  ask  them  before  the  testing  begins.  You  will  get  to  practice  before  testing  begins 
and  may  also  ask  questions  during  practice. 

Air  Commander 

The  United  States  has  just  learned  that  Ikestan  attacked  a  U.S.  base  in  the  middle  east.  The 
president  has  ordered  all  military  personnel  in  the  surrounding  areas  to  be  on  alert  for  possible 
attacks.  You  are  stationed  at  an  air  base  in  Tenya,  N.  Africa,  which  has  been  put  on  alert.  A  few 
hours  after  the  initial  attack  you  are  instructed  to  report  to  your  post  as  enemy  aircraft  are  making 
their  way  towards  your  base.  Aircraft  have  been  sent  to  intercept  and  destroy  these  incoming 
planes.  It  is  your  job  to  protect  your  base  from  enemy  aircraft  strikes  while  allowing  your  planes  to 
land  safely. 

Your  success  is  determined  by  the  number  of  reward  and  penalty  points  you  accrue.  You  will  want 
to  maximize  your  reward  points  and  minimize  your  penalty  points.  In  order  to  do  this,  you  will 
need  to  determine  which  aircraft  are  friendly  and  which  are  enemy  and  what  type  of  aircraft  they 
are.  At  the  same  time,  you  will  need  to  prioritize  incoming  targets  based  upon  their  range  from 
your  home  base  and  speed  to  determine  which  aircraft  are  the  most  critical  at  any  one  time.  You 
will  request  aircraft  identity  information  from  the  Intel  officer  (who  is  at  an  airbase  in  Western 
Europe)  to  support  this  prioritization  decision. 
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Point  assignments  have  been  made  for  each  type  of  aircraft  representing  the  reward  points  for 
destroying  the  aircraft  and  the  penalty  points  for  allowing  the  aircraft  to  land  at  your  home  base. 

The  points  are  based  on  the  mission  relevance  and  lethality  of  the  aircraft.  For  example,  transports 
can  carry  many  personnel  and  therefore  could  present  a  great  threat  if  they  land  at  a  U.S.  base,  while 
the  loss  of  a  U.S.  transport  would  be  a  devastating  blow  to  our  forces.  Friendly  aircraft  have  a  zero 
penalty  for  getting  through  to  home  base  (you  would  want  them  to  safely  land  at  home)  and  a 
negative  reward  associated  with  destroying  it  (you  would  not  want  to  destroy  your  own  planes). 
Enemy  aircraft,  however,  have  both  positive  reward  and  penalty  points. 

In  addition,  it  is  possible  for  some  targets  to  collide  with  one  another.  If  two  friendly  aircraft 
collide,  no  points  are  accrued  for  this.  If  two  enemy  aircraft  collide,  the  highest  reward  points  of  the 
two  aircraft  are  given.  However,  if  a  friendly  and  an  enemy  aircraft  collide,  you  receive  double  the 
penalty  points  of  the  enemy  aircraft.  Therefore,  it  is  in  your  best  interest  to  destroy  enemy  aircraft 
before  they  collide  with  friendly  planes. 

You  will  see  6  different  types  of  aircraft:  friendly  fighters,  enemy  fighters,  friendly  bombers,  enemy 
bombers,  friendly  transports  and  enemy  transports.  There  are  a  total  of  16  different  kinds  of  aircraft 
that  are  in  use  by  friendly  and  enemy  forces.  Below  is  a  table  of  these  aircraft,  their  color 
appearance  on  your  radar  screen  once  identified,  and  their  associated  reward  and  penalty  points. 


Plane  Categories 

Friendly  Fighters 

Types 

F/A-18 

F-15E 

F-16 

Color 

Blue 

Blue 

Blue 

Reward 

Points 

-20 

-40 

-60 

Penalty  Points 

0 

0 

0 

Friendly  Bombers 

B-52 

Green 

-50 

0 

B-l 

Green 

-80 

0 

B-2 

Green 

-100 

0 

Friendly  Transports 

C-130J 

Turquoise 

-120 

0 

C-21 

Turquoise 

-140 

0 

KC-135 

Turquoise 

-150 

0 

Enemy  Fighters 

Mig-29 

Red 

60 

10 

Su-35 

Red 

80 

20 

Su-37 

Red 

100 

10 

Enemy  Bombers 

Tu-22M 

Orange 

10 

50 

Tu-168 

Orange 

20 

60 

Enemy  Transports 

An- 124 

Yellow 

50 

50 

An-225 

Yellow 

60 

60 

You  will  view  the  aircraft  on  a  radar  screen,  similar  to  that  shown  on  the  following  page.  Your 
home  base  is  at  the  center  of  the  screen.  The  aircraft  will  appear  from  outside  the  radar  and  travel 
inward  towards  your  home  base.  It  is  your  job  to  let  the  friendly  aircraft  land  at  the  base,  but  the 
enemy  aircraft  must  be  destroyed  before  reaching  the  base.  An  aircraft  will  get  through  if  left  alone 
(it  will  land  on  its  own). 
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You  destroy  aircraft  by  launching  missiles  at  them.  There  are  two  types  of  missiles  available;  a 
Sparrow  which  is  smaller  and  faster  and  used  for  destroying  fighters  and  AMRAAM  (advanced 
medium  range  air  to  air  missiles)  which  are  long  range  radar  guided  missiles  used  for  destroying 
bombers  and  transports.  To  launch  missiles  at  an  aircraft,  use  the  computer  mouse  to  click  on  it 
once;  the  left  mouse  button  launches  Sparrows  and  the  right  mouse  button  launches  AMRAAMs. 
This  gives  the  command  to  launch  missile  defense  resources  against  the  target.  The  target  will  turn 
a  light  violet  color  to  indicate  that  it  has  been  targeted.  If  you  launch  an  inappropriate  missile  at  an 
aircraft  the  plane  may  not  be  destroyed  and  you  will  have  to  try  again. 

AW  ACS,  also  known  as  reconnaissance  planes,  will  also  be  present  on  your  screen.  They  are  small 
blue  circles  labeled  A,  B,  or  C.  It  is  their  job  to  determine  plane  identifications  and  relay  this 
information  to  the  Intel  Officer.  The  reliability  of  the  AW  AC  information  is  dependent  upon  their 
location,  the  closer  the  plane  to  home  base  (center  of  the  grid)  the  more  reliable  the  information.  At 
5  to  10  miles  out  (first  two  rings),  the  reliability  is  75%  at  15  to  20  miles  the  reliability  is  50%,  and 
at  25  to  30  miles  the  reliability  is  25%. 

Initially,  all  aircraft  appear  as  white  squares.  Below  each  square  is  the  aircraft  target  number  and  its 
speed.  In  order  for  you  to  determine  what  type  of  aircraft  is  on  your  radar,  you  must  request  this 
information  from  the  intelligence  officer.  This  is  done  by  typing  the  aircraft’s  number  in  the  request 
box  in  the  middle  left  side  of  the  radar  screen.  All  you  need  to  do  is  type  in  each  number  followed 
by  the  return  key.  The  Intel  officer  will  receive  and  process  your  information.  It’s  identity  will  be 
shown  by  the  color  of  the  square  on  your  radar  screen  (ex.  blue  for  friendly  fighters).  The  aircraft 
number  will  remain  the  same,  but  the  penalty  points  for  the  identification  that  has  been  made  will 
appear  below  the  target  number.  This  may  or  may  not  be  correct  depending  on  the  accuracy  of  the 
ID.  You  must  decide  what  to  do  with  each  aircraft,  either  processing  it  (destroying  it )  or  letting  it 
land  at  the  base.  At  all  times  during  this  task,  your  total  reward  and  penalty  points  are  shown  at  the 
top  left  comer  of  your  radar  screen. 

Once  the  Intel  Officer  has  sent  over  his  classification  for  the  aircraft,  you  will  receive  the  AWAC’s 
identification  for  that  particular  aircraft.  This  information  will  appear  in  the  lower  left  section  of 
your  screen  below  the  requested  information  box.  The  plane  will  be  identified  by  its  target  number 
and  the  AWAC’s  identification;  labeled  Source  A,  Source  B,  and  Source  C. 

Aircraft  will  continually  appear  on  your  radar  screen  until  the  testing  time  is  up.  Thus,  you  could 
potentially  process  hundreds  of  planes  in  the  6  minutes  allotted  for  each  session.  Please  be  aware 
that  this  task  moves  quickly.  We  want  you  to  work  as  quickly  and  as  accurately  as  you  can.  These 
instructions  may  seem  a  bit  confusing,  so  we  will  start  with  several  training  sessions  to  help  clarify 
your  task. 
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APPENDIX  E  :  ANOVA  Results 


AIR  COMMANDER 

LEVELS  ENCOUNTERED  DURING  PROCESSING  ARE: 

DISPLAY 

1.000  2.000  3.000 

WORKLOAD 

3.000  6.000  9.000 

(Sum  of  Actual  Reward  Points) 

DEP  VAR:  Actual  Reward  N:  162  MULTIPLE  R:  0.804  SQUARED  MULTIPLE  R:  0.646 
ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

1742979.012 

2 

871489.506 

8.680 

0.000 

WORKLOAD 

DISPLAY 

251666E+08 

2 

.125833E+08 

125.335 

0.000 

^WORKLOAD 

1139891.358 

4 

284972.840 

2.838 

0.026 

ERROR 

.153608E+08 

153 

100397.095 

(Sum  of  Actual  Penalty  Points) 

DEP  VAR:  Actual  Penalty  N:  162  MULTIPLE  R:  0.848  SQUARED  MULTIPLE  R:  0.720 
ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

376470.370 

2 

188235.185 

5.945 

0.003 

WORKLOAD 

.116314E+08 

2 

5815696.296 

183.682 

0.000 

DISPLAY 

*  WORKLOAD 

431481.481 

4 

107870.370 

3.407 

0.011 

ERROR 

4844255.556 

153 

31661.801 

(Adjusted  Reward  Points  0  - 

100) 

DEP  VAR:  Normalized  Reward  N:  162  MULTIPLE  R:  0.430 

SQUARED  MULTIPLE  R:  0.185 

ANALYSIS  OF  VARIANCE 

SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

87.698 

2 

243.849 

3.170 

0.045 

WORKLOAD 

1822.278 

2 

911.139 

11.844 

0.000 

DISPLAY 

*  WORKLOAD 

359.400 

4 

89.850 

1.168 

0.327 

ERROR 

11770.177 

153 

76.929 
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(Adjusted  Penalty  Points  0-100) 

DEP  VAR:Normalized  Penalty  N:  162  MULTIPLE  R:  0.770  SQUARED  MULTIPLE  R:  0.593 


ANALYSIS  OF  VARIANCE 


SOURCE 


SUM-OF-SQUARES  DF  MEAN-SQUARE  F-RATIO 


DISPLAY  2525.178 

WORKLOAD  32809.802 

DISPLAY 

*  WORKLOAD  2846.997 


2 

1262.589 

7.359 

2 

16404.901 

95.619 

4 

711.749 

4.149 

ERROR 


26249.491  153  171.565 


P 

0.001 

0.000 

0.003 


(Categorization  time  for  attacked  targets) 

DEP  VAR:  Categorize  to  KILL  N:  155  MULTIPLE  R:  0.554  SQUARED  MULTIPLE  R:  0.307 
ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

82.351 

2 

41.176 

10.108 

0.000 

WORKLOAD 

DISPLAY 

174.646 

2 

87.323 

21.437 

0.000 

*  WORKLOAD 

6.887 

4 

1.722 

0.423 

0.792 

ERROR 

594.720 

146 

4.073 

(Categorization  time  for  collided  targets) 

DEP  VAR: Categorize  to  COLLIDE  N:  102 

MULTIPLE  R:  0.535  SQUARED  MULTIPLE  R:  0.286 

ANALYSIS  OF  VARIANCE 

SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

423.445 

2 

211.723 

2.539 

0.084 

WORKLOAD 

DISPLAY 

2030.438 

2 

1015.219 

12.176 

0.000 

^WORKLOAD 

499.917 

4 

124.979 

1.499 

0.209 

ERROR 

7754.338 

93 

83.380 

(Categorization  time  for  landed  targets) 

DEP  VAR:  Categorize  to  LAND  N:  156  MULTIPLE  R:  0.89 1 

SQUARED  MULTIPLE  R:  0.793 

ANALYSIS  OF  VARIANCE 

SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

3437.595 

2 

1718.798 

19.001 

0.000 

WORKLOAD 

DISPLAY 

46823.232 

2 

23411.616 

258.815 

0.000 

*  WORKLOAD 

123.091 

4 

30.773 

0.340 

0.850 

ERROR 

13297.176 

147 

90.457 
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INTELLIGENCE  OFFICER 


LEVELS  ENCOUNTERED  DURING  PROCESSING  ARE: 
COVERAGE 

1.000  2.000  3.000 

POSITION 

1.000  2.000  3.000  4.000 


DEP  VAR:  VIEW  TIME  N:  216  MULTIPLE  R:  0.344  SQUARED  MULTIPLE  R:  0. 1 1 8 
ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

COVERAGE 

14.407 

2 

7.203 

12.824 

0.000 

POSITION 

0.632 

3 

0.211 

0.375 

0.771 

COVERAGE 

♦POSITION 

0.359 

6 

0.060 

0.106 

0.996 

ERROR 

114.585 

204 

0.562 

DEP  VAR:CORRECT  TIME  N:  21 6  MULTIPLE  R:  0.834 

SQUARED  MULTIPLE  R:  0.696 

ANALYSIS  OF  VARIANCE 

SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

COVERAGE 

18.889 

2 

9.444 

167.674 

0.000 

POSITION 

COVERAGE 

3.057 

3 

1.019 

18.091 

0.000 

♦POSITION 

4.400 

6 

0.733 

13.020 

0.000 

ERROR 

11.491 

204 

0.056 

DEP  VAR:  VIEWTIME  N:  216  MULTIPLE  R:  0.977  SQUARED  MULTIPLE  R:  0.954 
ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

SUBJECT 

93.945 

17 

5.526 

94.178 

0.000 

COVERAGE 

14.407 

2 

7.203 

122.760 

0.000 

POSITION 

COVERAGE 

0.632 

3 

0.211 

3.590 

0.016 

♦POSITION 

SUBJECT 

0.359 

6 

0.060 

1.018 

0.418 

♦COVERAGE 

SUBJECT 

9.363 

34 

0.275 

4.693 

0.000 

♦POSITION 

ERROR 

5.292 

5.985 

51 

102 

0.104 

0.059 

1.768 

0.008 
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DEP  VARiCORRECT  TIME  N:  216  MULTIPLE  R:  0.955  SQUARED  MULTIPLE  R:  0.91 1 


ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

SUBJECT 

4.751 

17 

0.279 

8.501 

0.000 

COVERAGE 

18.889 

2 

9.444 

287.285 

0.000 

POSITION 

COVERAGE 

3.057 

3 

1.019 

30.997 

0.000 

♦POSITION 

SUBJECT 

4.400 

6 

0.733 

22.308 

0.000 

♦COVERAGE 

SUBJECT 

1.743 

34 

0.051 

1.559 

0.046 

♦POSITION 

ERROR 

1.644 

3.353 

51 

102 

0.032 

0.033 

0.980 

0.522 

LEVELS  ENCOUNTERED  DURING  PROCESSING  ARE: 
DISPLAY 

1.000  2.000  3.000 

WORKLOAD 

3.000  6.000  9.000 

10  CASES  DELETED  DUE  TO  MISSING  DATA. 


(Mean  view  time) 

DEP  VAR:  VIEW  TIME  N:  152  MULTIPLE  R:  0.672  SQUARED  MULTIPLE  R:  0.451 
ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

21.120 

2 

10.560 

8.821 

0.000 

WORKLOAD 

DISPLAY 

108.197 

2 

54.099 

45.190 

0.000 

♦WORKLOAD 

15.426 

4 

3.856 

3.221 

0.014 

ERROR 

171.190 

143 

1.197 

(Mean  request  time) 

DEP  VAR:REQUESTT 

N:  78  MULTIPLE  R:  0.364  SQUARED 

MULTIPLE  R:  0.132 

ANALYSIS  OF  VARIANCE 

SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

158.485 

2 

79.243 

2.822 

0.066 

WORKLOAD 

DISPLAY 

62.682 

2 

31.341 

1.116 

0.333 

♦WORKLOAD 

44.148 

4 

11.037 

0.393 

0.813 

ERROR 

1937.224 

69 

28.076 
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(Percent  requested  transformed) 

DEP  V AR : AREQUEST  N:  158  MULTIPLE  R:  0.567  SQUARED  MULTIPLE  R:  0.321 
ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

7.896 

2 

3.948 

20.660 

0.000 

WORKLOAD 

DISPLAY 

5.546 

2 

2.773 

14.511 

0.000 

♦WORKLOAD 

0.838 

4 

0.209 

1.096 

0.361 

ERROR 

28.474 

149 

0.191 

(Percent  on  top  list  transformed) 

DEP  VAR:  ATOP 

N:  158  MULTIPLE  R:  0.544  SQUARED  MULTIPLE  R:  0.296 

ANALYSIS  OF  VARIANCE 

SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

DISPLAY 

2.134 

2 

1.067 

10.832 

0.000 

WORKLOAD 

DISPLAY 

3.454 

2 

1.727 

17.529 

0.000 

♦WORKLOAD 

1.054 

4 

0.263 

2.674 

0.034 

ERROR 

14.678 

149  0.099 
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GLOSSARY 


AMRAAM 

ANOVA 

AWACS 

SA 


Advanced  Medium  Range  Air  to  Air  Missile 
Analysis  of  Variance 
Airborne  Warning  and  Control  System 
Situation  Awareness 
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